Imaging unit, imaging apparatus, and computer readable medium storing thereon an imaging control program

ABSTRACT

When the amplification ratio is low and strong incident light causes a large charge, the signal retrieved from regions where the incident light is weak is also weak, but when the amplification ratio is high in regions where the incident light is weak, the signal retrieved from regions where the incident light is strong becomes saturated. Therefore, the dynamic range of the imaging unit is narrow. Provided is an imaging unit comprising an imaging section that includes a first group having one or more pixels and a second group having one or more pixels different from those of the first group; and a control section that, while a single charge accumulation is performed in the first group, causes pixel signals to be output by performing charge accumulation in the second group a number of times differing from a number of times charge accumulation is performed in the first group.

This is a Continuation of U.S. application Ser. No. 17/371,751 filedJul. 9, 2021, which is a Continuation of U.S. application Ser. No.16/844,231 filed Apr. 9, 2020, which is a Continuation of U.S.application Ser. No. 15/946,168 filed Apr. 5, 2018, which is aContinuation of U.S. application Ser. No. 15/401,683 filed Jan. 9, 2017,which is a Continuation of U.S. application Ser. No. 14/500,030 filedSep. 29, 2014, which is a Continuation of International Application No.PCT/JP2013/002148 filed Mar. 28, 2013, which claims the benefit ofJapanese Application No. 2012-128092 filed Jun. 5, 2012 and JapaneseApplication No. 2012-082312 filed Mar. 30, 2012. The disclosures of theprior applications are hereby incorporated by reference herein in theirentireties.

BACKGROUND 1. Technical Field

The present invention relates to an imaging unit, an imaging apparatus,and a computer readable medium storing thereon an imaging controlprogram.

2. Related Art

An imaging unit is known in which a back emission type imaging chip anda signal processing chip are connected via micro-bumps provided for eachcell containing a group of pixels.

-   Patent Document 1: Japanese Patent Application Publication No.    2006-49361

When the amplification ratio is low in a case where the incident lightis strong and there is a large amount of charge, the signal retrievedfrom the regions where the incident light is weak is also weak. On theother hand, when the amplification ratio is high in regions where theincident light is weak, the signal retrieved from regions where theincident light is strong becomes saturated. Therefore, the dynamic rangeof the imaging unit is limited to a narrow range.

SUMMARY

According to a first aspect of the present invention, provided is animaging unit comprising an imaging section that includes a first grouphaving one or more pixels and a second group having one or more pixelsthat are different from the one or more pixels of the first group; and acontrol section that, while a single charge accumulation is beingperformed in the first group, causes respective pixel signals to beoutput by performing charge accumulation in the second group a number oftimes differing from a number of times charge accumulation is performedin the first group.

According to a second aspect of the present invention, provided is animaging apparatus including the imaging unit described above.

According to a third aspect of the present invention, provided is animaging control program that, when executed, causes a computer toperform first initiation of beginning charge accumulation in a firstgroup including one or more pixels; second initiation of beginningcharge accumulation in a second group including one or more pixels,which are different from the pixels of the first group; second outputof, before or at the moment when the charge accumulation in the firstgroup ends, ending the charge accumulation in the second group andoutputting a pixel signal; and first output of, after repeating thesecond initiation and the second output a plurality of times, ending thecharge accumulation in the first group and outputting a pixel signal.

According to a fourth aspect of the present invention, provided is animaging apparatus comprising an imaging section including a first grouphaving one or more pixels and a second group having one or more pixelsthat are different from the one or more pixels of the first group; acontrol section that, while a plurality of charge accumulations arebeing performed in the first group, causes respective pixel signals tobe output by performing a plurality of charge accumulations in thesecond group; and a calculating section that performs computation suchthat a process applied to the pixel signals output from the first groupdiffers from a process applied to the pixel signals output from thesecond group.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a cross-sectional view of a back emission image MOS capturingelement according to an embodiment of the present invention.

FIG. 2 is a view for describing the single group and the pixelarrangement of the imaging chip.

FIG. 3 is a circuit diagram corresponding to a single group of theimaging chip.

FIG. 4 is a block diagram showing a functional configuration of theimaging element.

FIG. 5 is a block diagram showing a configuration of the imagingapparatus according to the present embodiment.

FIGS. 6A and 6B are view for describing an exemplary scene and regionallocation.

FIG. 7 is used to describe the charge accumulation in each dividedregion.

FIG. 8 shows the relationship between the number of integrations and thedynamic range.

FIG. 9 is a flow showing processing of the image capturing operations.

FIG. 10 is a block diagram showing a detailed configuration as anexample of the signal processing chip.

FIG. 11 is a view describing the flow of pixel signals from the imagingchip to the signal processing chip.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, some embodiments of the present invention will bedescribed. The embodiments do not limit the invention according to theclaims, and all the combinations of the features described in theembodiments are not necessarily essential to means provided by aspectsof the invention.

FIG. 1 is a cross-sectional view of a back emission imaging element 100according to an embodiment of the present invention. The imaging element100 includes an imaging chip 113 that outputs a pixel signalcorresponding to incident light, a signal processing chip 111 thatprocesses the pixel signal, and a memory chip 112 that records the pixelsignals. The imaging chip 113, the signal processing chip 111, and thememory chip 112 are layered, and are electrically connected to eachother by conductive bumps 109 made of Cu, for example.

As shown in the drawings, the incident light is mainly incident in thepositive Z-axis direction, as shown by the hollow white arrow. In thepresent embodiment, the surface of the imaging chip 113 to which theincident light is incident is referred to as the back surface.Furthermore, as shown by the axes, the direction to the left in theplane of the drawing and orthogonal to the Z axis is the positive X-axisdirection, and the direction coming forward from the plane of thedrawing and orthogonal to the X axis and Z axis is the positive Y-axisdirection. In many of the following drawings, the coordinate axes ofFIG. 1 are used as a reference, and these coordinate axes are includedin the drawings to indicate the orientation.

The imaging chip 113 is a back emission MOS image sensor, for example.The PD layer 106 is arranged on the back surface side of the wiringlayer 108. The PD layer 106 includes a plurality of PDs (photodiodes)104 arranged two-dimensionally, and transistors 105 providedcorresponding to the PDs 104.

Color filters 102 are provided on the incident light side of the PDlayers 106, with a passivation film 103 arranged therebetween. The colorfilters 102 include a plurality of types that pass different wavelengthregions, and have a prescribed arrangement corresponding respectively tothe PDs 104. The arrangement of the color filters 102 is describedfurther below. Each set of a color filter 102, a PD 104, and atransistor 105 forms one pixel.

Microlenses 101 are arranged on the incident light side of the colorfilter 102, corresponding respectively to the pixels. The microlenses101 each gather the incident light toward the corresponding PD 104.

The wiring layer 108 includes wires 107 that transmit the pixel signalsfrom the PD layer 106 to the signal processing chip 111. The wire 107may be multilayer, and may include active elements and passive elements.

A plurality of bumps 109 are arranged on the front surface of the wiringlayer 108. These bumps 109 are aligned with a plurality of bumps 109 onthe surface facing the signal processing chip 111, and the aligned bumps109 are connected to each other to form an electrical connection, bypressing the imaging chip 113 and the signal processing chip 111together, for example.

In the same manner, a plurality of bumps 109 are arranged on thesurfaces of the signal processing chip 111 and the memory chip 112facing each other. The These bumps 109 are aligned with each other andthe aligned bumps 109 are connected to each other to form an electricalconnection, by pressing the signal processing chip 111 and the memorychip 112 together, for example.

The connection between the bumps 109 is not limited to a Cu bumpconnection through solid phase diffusion, and a micro-bump connectionthrough solder fusion may be adopted instead. Furthermore, approximately1 bump 109 should be provided for each pixel group, for example, asdescribed further below. Accordingly, the bumps 109 may be larger thanthe pitch of the PDs 104. Furthermore, in the peripheral region outsideof the pixel regions where the pixels are arranged, bumps that arelarger than the bumps 109 corresponding to the pixel regions may beprovided.

The signal processing chip 111 includes a TSV (Through Silicon Via) 110that connects each of the circuits provided on the front and backsurfaces. The TSV 110 is preferably provided in the peripheral region.Furthermore, the TSV 110 may be provided both in the peripheral regionof the imaging chip 113 and the memory chip 112.

FIG. 2 is a view for describing the single group 131 and the pixelarrangement of the imaging chip 113. In particular, the imaging chip 113is shown as seen from the back surface side. Twenty million or morepixels are arranged in the pixel region in a matrix formation. In thepresent embodiment, 16 pixels in a 4×4 arrangement of adjacent pixelsform one group. The grid lines in the drawing indicate single groups 131formed by grouping adjacent pixels.

As shown in the enlarged view of a portion of the pixel region, a singlegroup 131 has a so-called Bayer arrangement in which green pixels Gb andGr, a blue pixel B, and a red pixel R are arranged in respectivecorners. The green pixels are pixels that have green filters as thecolor filters 102, and receive light in the green wavelength band withinthe incident light. Similarly, the blue pixel is a pixel that has a bluefilter as the color filter 102 and receives light in the blue wavelengthband, and the red pixel is a pixel that has a red filter as the colorfilter 102 and receives light in the red wavelength band.

FIG. 3 is a circuit diagram corresponding to a single group 131 of theimaging chip 113. In the drawing, the rectangle surrounded by the dottedline is representative of a circuit corresponding to one pixel. At leastsome of the transistors in the following description correspond to thetransistor 105 shown in FIG. 1 .

As described above, the single group 131 is formed from 16 pixels. The16 PDs 104 corresponding respectively to the pixels are each connectedto the transfer transistor 302, and a TX wire 307 that supplies atransfer pulse is connected to each gate of each transfer transistor302. In the present embodiment, the TX wire 307 is connected in a mannerto be shared by 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source ofthe corresponding reset transistor 303, and a so-called floatingdiffusion FD between the drain of each transfer transistor 302 and thesource of the corresponding reset transistor 303 is connected to thegate of the corresponding amplification transistor 304. The drain of thereset transistor 303 is connected to the Vdd wire 310 that is suppliedwith the power supply voltage, and the gate of the reset transistor 303is connected to the reset wire 306 that is supplied with a reset pulse.In the present embodiment, the reset wire 306 is connected in a mannerto be shared by 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to the Vddwire 310 that is supplied with the power supply voltage. The source ofeach amplification transistor 304 is connected to the drain of thecorresponding selection transistor 305. Each gate of a selectiontransistor is connected to the decoder wire 308 that is supplied with aconnection pulse. In the present embodiment, the decoder wire 308 isprovided independently for each set of 16 selection transistors 305. Thesource of each selection transistor 305 is connected to a common outputwire 309. The negative current source 311 supplies current to the outputwire 309. In other words, the output wire 309 for a selection transistor305 is formed by a source follower. The negative current source 311 maybe provided on the imaging chip 113 side, or on the signal processingchip 111 side.

The following describes the flow from the start of charge accumulationto the pixel output after ending the accumulation. The reset pulse isapplied to the reset transistor 303 through the reset wire 306, and atthe same time the transfer pulse is applied to the transfer transistor302 through the TX wire 307, thereby resetting the potentials of the PD104 and the floating diffusion FD.

When the application of the transfer pulse ends, the PD 104 converts thereceived incident light into charge and accumulates this charge. Afterthis, when the transfer pulse is again applied in a state where thereset pulse is not being applied, the accumulated charge is transferredto the floating diffusion FD, and the potential of the floatingdiffusion FD changes from the reset potential to a signal potentialafter charge accumulation. When the selection pulse is applied to theselection transistor 305 through the decoder wire 308, the change of thesignal potential of the floating diffusion FD is communicated to theoutput wire 309 through the amplification transistor 304 and theselection transistor 305. In this way, the pixel signal corresponding tothe reset potential and the signal potential is output to the outputwire 309 from the single pixel.

As shown in the drawings, in the present embodiment, the reset wire 306and the TX wire 307 are shared by 16 pixels forming a single group 131.In other words, each reset pulse and transfer pulse is appliedsimultaneously to all 16 pixels. Accordingly, all of the pixels forminga single group 131 begin accumulating charge at the same timing, andfinish accumulating charge at the same timing. However, the pixelsignals corresponding to the accumulated charges are selectively outputto the output wire 309 as a result of the respective selectiontransistors 305 sequentially applying selection pulses.

In this way, by configuring the circuit with single groups 131 as astandard, the charge accumulation time of each single group 131 can becontrolled. In other words, adjacent single groups 131 can outputrespective pixel signals with different charge accumulation times. Asyet another rephrasing, while one single group 131 is performing asingle charge accumulation, another single group 131 can repeatedlyperform any number of charge accumulations, and output a correspondingpixel signal. The specific output control is described below.

FIG. 4 is a block diagram showing a functional configuration of theimaging element 100. The analog multiplexer 411 sequentially selects PDs104 among the 16 PDs 104 forming the single group 131, and causes eachPD 104 to output a pixel signal to the output wire 309. The multiplexer411 is formed in the imaging chip 113 along with the PDs 104.

The pixel signal output via the multiplexer 411 undergoes CDS(Correlated Double Sampling) and an A/D (Analog/Digital) conversion bythe signal processing circuit 412 that performs CDS and A/D conversionsformed in the signal processing chip 111. The pixel signal resultingfrom the A/D conversion is handed over to the de-multiplexer 413, andstored in the pixel memory 414 corresponding to the pixel. Each pixelmemory 414 has a capacity that enables storage of a number of pixelsignals corresponding to a maximum number of integrations, which isdescribed further below. The de-multiplexer 413 and the pixel memories414 are formed in the memory chip 112.

The A/D conversion includes converting the input analog pixel signalinto a 12-bit digital pixel signal. At the same time, the signalprocessing circuit 412 attaches a 3-bit index number corresponding tothe number of integrations, which is described further below, and handsover the digital pixel signal having a total size of 15 bits to thede-multiplexer 413. Accordingly, the pixel memory 414 stores a 15-bitdigital pixel signal corresponding to one charge accumulation.

The calculating circuit 415 processes the pixel signal stored in thepixel memory 414, and hands the resulting signal over to the imageprocessing section provided at a later stage. The calculating circuit415 may be provided in the signal processing chip 111, or may beprovided in the memory chip 112. The drawings show the connection to onegroup, but a calculating circuit 415 is actually provided for eachgroup, and the calculating circuits 415 operate in parallel. It shouldbe noted that a calculating circuit 415 need not be provided for eachgroup, and instead a single calculating circuit 415 may performsequential processing while sequentially referencing the values in thepixel memories 414 corresponding to respective groups, for example.

FIG. 5 is a block diagram showing a configuration of the imagingapparatus according to the present embodiment. The imaging apparatus 500includes an imaging lens 520 serving as an imaging optical system, andthe imaging lens 520 guides subject light that is incident along theoptical axis OA to the imaging element 100. The imaging lens 520 may bean exchangeable lens that can be attached to and detached from theimaging apparatus 500. The imaging apparatus 500 primarily includes theimaging element 100, a system control section 501, a driving section502, a photometric section 503, a work memory 504, a storage section505, and a display section 506.

The imaging lens 520 is formed by a plurality of optical lens groups,and focuses the subject light from a scene near a focal plane. In FIG. 1, the imaging lens 520 is represented by a single virtual lens arrangednear the pupil. The driving section 502 is a control circuit thatperforms charge accumulation control such as region control and timingcontrol of the imaging element 100, according to instructions from thesystem control section 501. In this sense, the driving section 502 canbe said to have the function of an imaging element control section thatcauses the pixel signals to be output by performing charge accumulationfor the imaging element 100. The driving section 502 forms an imagingunit when combined with the imaging element 100. The control circuitforming the driving section 502 may be formed as a chip and layered onthe imaging element 100.

The imaging element 100 hands the pixel signal over to the imageprocessing section 511 of the system control section 501. The imageprocessing section 511 applies various types of image processing, withthe work memory 504 as a work space, to form image data. For example,when generating image data in a JPEG file format, the image processingsection 511 performs a white balance process, a gamma process, and thelike, and then performs a compression process. The generated image datais recorded in the storage section 505, and is converted to a displaysignal to be displayed in the display section 506 during a predeterminedtime.

The photometric section 503 detects the brightness distribution of ascene before the imaging sequence for generating the image data. Thephotometric section 503 includes an AE sensor with approximately 1million pixels, for example. The calculating section 512 of the systemcontrol section 501 receives the output of the photometric section 503and calculates the brightness of each region of the system controlsection 501. The calculating section 512 determines the shutter speed,diaphragm value, and ISO sensitivity according to the calculatedbrightness distribution. In the present embodiment, the calculatingsection 512 further determines how many times the charge accumulation isrepeated in each pixel group region of the imaging chip 113, until thedetermined shutter speed is reached. The calculating section 512performs a variety of calculations for operating the imaging apparatus500.

FIGS. 6A and 6B are views for describing an exemplary scene and regionallocation. FIG. 6A shows a scene captured by pixel regions of theimaging chip 113. Specifically, the scene includes an intermediatesubject 602 and a shadowed subject 601 included in the room environmentand a highlighted subject 603 of the outdoor environment seen within theframe 604. In this way, when capturing an image of a scene in whichthere is a large brightness difference from the highlight portion to theshadow portion, if a conventional imaging element is used, underexposureoccurs in the shadow portion if charge accumulation is performed withthe highlight portion as the standard, and overexposure occurs in thehighlight portion if charge accumulation is performed with the shadowportion as the standard. In other words, the dynamic range of thephotodiodes for a scene with a large brightness difference isinsufficient to output an image signal in which charge accumulation isuniform for both the highlight portion and the shadow portion.Therefore, in the present invention, the dynamic range is substantiallyenlarged by dividing the scene into the partial regions of the highlightportion and the shadow portion, and creating differences in the numberof charge accumulations of the photodiodes corresponding to therespective portions.

FIG. 6B shows the region division for the pixel region of the imagingchip 113. The calculating section 512 analyzes the scene of FIG. 6Acaptured by the photometric section 503, and divides the pixel regionusing brightness as a reference. For example, the system control section501 performs scene acquisition a plurality of times while changing theexposure time for the photometric section 503, and the calculatingsection 512 determines the dividing lines of the pixel regions byreferencing the change in the distribution of the overexposure regionand the underexposure region. In the example of FIG. 6B, the calculatingsection 512 divides the scene into three regions, which are the shadowregion 611, the intermediate region 612, and the highlight region 613.

The dividing lines are defined along the boundaries of the single groups131. In other words, each of the divided regions includes an integernumber of groups. Furthermore, the pixels of each group contained in thesame region output the same number of pixel signals and perform chargeaccumulation the same number of times within the period corresponding tothe shutter speed determined by the calculating section 512. Ifassociated regions are different, the number of pixel signals output andthe number of times charge accumulation is performed are different.

FIG. 7 is used to describe the charge accumulation in each regiondivided as shown in FIGS. 6A and 6B. Upon receiving image capturingpreparation instructions from the user, the calculating section 512determines the shutter speed (exposure time) TO from the output of thephotometric section 503. Furthermore, the shadow region 611, theintermediate region 612, and the highlight region 613 are divided asdescribed above, and the number of charge accumulations is determinedfrom the brightness information of each region. The number of chargeaccumulations is determined such that pixel saturation does not occurfrom one charge accumulation. For example, the number of chargeaccumulations may be determined using a reference that is 80% to 90% ofthe charge that can be accumulated in a single charge accumulationoperation.

Here, charge accumulation is performed once for the shadow region 611.In other words, the charge accumulation time matches the determinedexposure time T0. Furthermore, the charge accumulation is performedtwice for the intermediate region 612. In other words, the chargeaccumulation is performed twice during the exposure time T0, with eachcharge accumulation time being T0/2. Furthermore, the chargeaccumulation is performed four times for the highlight region 613. Inother words, the charge accumulation is performed four times during theexposure time T0, with each charge accumulation time being T0/4.

Upon receiving image capturing instructions from the user at the timet=0, the driving section 502 applies a reset pulse and a transfer pulseto the pixels of the groups associated with one of the regions. Theapplication of these pulses triggers the start of charge accumulationfor one of the pixels.

At the time t=T0/4, the driving section 502 applies the transfer pulseto the pixels of the group associated with the highlight region 613. Thedriving section 502 sequentially applies selection pulses to the pixelswithin each group, to cause each of the pixel signals to be output tothe output wire 309. When the pixel signals of all of the pixels in thegroup have been output, the driving section 502 again applies the resetpulse and the transfer pulse to the pixels of the group associated withthe highlight region 613, to begin the second charge accumulation.

Since time is needed for the pixel signal selection output, there is atime difference between the end of the first charge accumulation and thestart of the second charge accumulation. If this time difference isshort enough to be substantially ignored, the time obtained by dividingthe exposure time T0 by the number of charge accumulations should be setas the first charge accumulation time. On the other hand, if this timedifference cannot be ignored, then the exposure time T0 should beadjusted in consideration of this time difference, such that the firstcharge accumulation time is shorter than the time obtained by dividingthe exposure time T0 by the number of charge accumulations.

At the time t=T0/2, the driving section 502 applies the transfer pulseto the pixels of the groups associated with the intermediate region 612and the highlight region 613. The driving section 502 then sequentiallyapplies the selection pulses to the pixels in each of these groups, tocause the respective pixel signals to be output by the output wire 309.When the pixel signals of all of the pixels in these groups have beenoutput, the driving section 502 again applies the reset pulse and thetransfer pulse to the pixels in the groups associated with theintermediate region 612 and the highlight region 613, to begin thesecond charge accumulation for the intermediate region 612 and the thirdcharge accumulation for the highlight region 613.

At the time t=3T0/4, the driving section 502 applies the transfer pulseto the pixels of the group associated with the highlight region 613. Thedriving section 502 sequentially applies selection pulses to the pixelswithin each group, to cause each of the pixel signals to be output tothe output wire 309. When the pixel signals of all of the pixels in thegroup have been output, the driving section 502 again applies the resetpulse and the transfer pulse to the pixels of the group associated withthe highlight region 613, to begin the fourth charge accumulation.

At the time t=T0, the driving section 502 applies the transfer pulse tothe pixels in all regions. The driving section 502 sequentially appliesthe selection pulses to the pixels in each group, to cause each pixelsignal to be output to the output wire 309. With the control describedabove, pixel signals from one charge accumulation are stored in thepixel memories 414 corresponding to the shadow region 611, pixel signalsfrom two charge accumulations are stored in the pixel memories 414corresponding to the intermediate region 612, and pixel signals fromfour charge accumulations are stored in the pixel memories 414corresponding to the highlight region 613.

These pixel signals are transferred sequentially to the image processingsection 511. The image processing section 511 generates image data witha high dynamic range from these pixel signals. The detailed processingis described further below.

FIG. 8 shows the relationship between the number of integrations and thedynamic range. The pixel signals from a plurality of chargeaccumulations performed repeatedly undergo computational processing bythe image processing section 511 to form a portion of the image datawith the high dynamic range.

When the number of integrations is 1, i.e. when the dynamic range of aregion for which charge accumulation is performed once is used as areference, the enlargement of the dynamic range of a region in which thenumber of integrations is 2, i.e. a region where the charge accumulationis performed twice and the output signals are integrated, is one stage.Similarly, the enlargement is two stages when the number of integrationsis 4, and is 7 stages when the number of integrations is 128. In otherwords, in order to enlarge the dynamic range by n stages, the outputsignals should be integrated 2n times.

Here, in order to identify the number of times that the image processingsection 511 has performed charge accumulation in each of the dividedregions, the image signals are each provided with a 3-bit index numberindicating the number of integrations. As shown in the drawings, theindex numbers are allocated in a manner of 000 for one integration, 001for two integrations, . . . , and 111 for 128 integrations.

The image processing section 511 references the index number of eachpixel signal received from the calculating circuit 415, and if theresult of this referencing is that the number of integrations is two ormore, performs an integration process on the pixel signal. For example,when the number of integrations is two (one stage), for two pixelsignals, the image processing section 511 adds the top 11 bits of the12-bit pixel signal corresponding to the charge accumulation, therebygenerating one 12-bit signal. In the same manner, when the number ofintegrations is 128 (seven stages), for 128 pixel signals, the imageprocessing section 511 adds together the top 5 bits of the 12-bit pixelsignal corresponding to the charge accumulation, thereby generating one12-bit signal. In other words, a number of top bits equal to the numberof stages corresponding to the number of integrations subtracted from 12are added together, to generate one 12-bit pixel signal. The lower bitsthat are not used in the addition are deleted.

With this type of processing, it is possible to shift the brightnessrange having a gradation sequence to the high brightness side, inaccordance with the number of integrations. In other words, 12 bits areallocated for a limited range on the high brightness side. Accordingly,the image regions that conventionally experienced overexposure can beprovided with a gradation sequence.

However, since 12 bits are allocated to different brightness ranges inother divided regions, the image data cannot be generated simply byconnecting all of the regions. Therefore, in order to maintain theachieved gradation sequence as much as possible while generating 12-bitimage data for all of the regions, the image processing section 511performs a requantization process using the pixel with highestbrightness and the pixel with the lowest brightness as a reference.Specifically, the image processing section 511 performs quantization byapplying a gamma conversion, in a manner to smooth the gradationsequence. By performing this process, the image data with a high dynamicrange can be obtained.

The number of integrations is not limited to a case in which the pixelsignals are provided with 3-bit index numbers as described above, andmay be recorded in associated information other than the pixel signals.Furthermore, by omitting the index numbers from the pixel signals andinstead counting the number of pixel signals stored in the pixel memory414, the number of integrations may be obtained when performing theadding process.

In the image processing described above, the requantization process isperformed such that all regions fall within the 12-bit image data, butthe number of output bits may be increased according to the maximumnumber of integrations for the number of bits of the pixel signals. Forexample, if the maximum number of integrations is set to 16 (fourstages), 16-bit image data for all of the regions may be used for the12-bit pixel signals. With this processing, the image data can begenerated without losing digits.

The following describes a series of image capturing operations. FIG. 9is a flow showing processing of the image capturing operations. Thisflow is begun by turning ON the power supply of the imaging apparatus500.

At step S101, the system control section 501 stands by until the switchSW1 is pressed, which indicates the image capturing preparationinstructions. When pressing of the switch SW1 is detected, the processmoves to step S102.

At step S102, the system control section 501 performs a light measuringprocess. Specifically, the system control section 501 obtains the outputof the photometric section 503, and the calculating section 512calculates a brightness distribution of the scene. The process thenmoves to step S103, and the shutter speed, region division, number ofintegrations, and the like are calculated as described above.

When the image capturing preparation operation is finished, the processmoves to step S104 and the standby state continues until the switch SW2is pressed, which indicates the image capturing instructions. At thistime, if a time longer than a predetermined time Tw has passed (the YESof step S105), the process returns to step S101. If pressing of theswitch SW2 is detected before the time Tw has passed (the NO of stepS105), then the process moves to step S106.

At step S106, the driving section 502 that has received the instructionsof the system control section 501 performs the charge accumulationprocess and signal retrieval process described using FIG. 7 . Whenretrieval of all of the signals has finished, the process moves to stepS107, the image processing described using FIG. 8 is performed, and arecording process is performed to record the generated image data in thestorage section.

When the recording process is finished, the process moves to step S108,and a judgment is made as to whether the power supply of the imagingapparatus 500 has been turned OFF. If the power supply has not beenturned OFF the process returns to step S101, and if the power supply hasbeen turned OFF, the image capturing operation process is finished.

The following describes an exemplary detailed configuration of thesignal processing chip 111. FIG. 10 is a block diagram showing adetailed configuration as an example of the signal processing chip 111.In the description using FIG. 4 , an example was used in which thede-multiplexer 413 and the pixel memories 414 are formed in the memorychip 112, but the following is an example in which these components areformed in the signal processing chip 111.

The signal processing chip 111 fulfills the function of the drivingsection 502. The signal processing chip 111 includes a sensor controlsection 441 to share the control function, a block control section 442,a synchronization control section 443, a signal control section 444, anda drive control section 420 that performs overall control of the controlsections. The drive control section 420 converts the instructions fromthe system control section 501 into control signals that can be executedby each control section, and passes these control signals to the controlsections.

The sensor control section 441 performs transmission control to transmita control pulse relating to charge accumulation and charge retrieval ofeach pixel to the imaging chip 113. Specifically, the sensor controlsection 441 controls the start and end of the charge accumulation bytransmitting the reset pulse and the transfer pulse to each pixel, andcauses the pixel signals to be output to the output wire 309 bytransmitting the selection pulse to the pixels to be retrieved.

The block control section 442 transmits, to the imaging chip 113, anidentification pulse that identifies single groups 131 to be controlled.As described using FIG. 6 and the like, the divided regions include aplurality of single groups 131 that are adjacent to each other. Thesingle groups 131 that are associated with the same region form oneblock. The pixels contained in the same block all begin the chargeaccumulation at the same timing and end the charge accumulation at thesame timing. The block control section 442 performs the role of formingthe single groups 131 in blocks, by transmitting the identificationpulse to the single groups 131 that are targets based on theinstructions from the drive control section 420. The transfer pulse andthe reset pulse received by each pixel via the TX wire 307 and the resetwire 306 are the AND of respective pulses transmitted from the sensorcontrol section 441 and the identification pulse transmitted by theblock control section 442. In this way, by controlling the regions inblocks that are independent from each other, the charge accumulationcontrol described using FIG. 7 is realized. The block instructions fromthe drive control section are described in detail further below.

The synchronization control section 443 transmits a synchronizationsignal to the imaging chip 113. Each pulse is synchronized with thesynchronization signal and active in the imaging chip 113. For example,by adjusting the synchronization signal, random control, thinningcontrol, and the like are realized targeting only the identified pixelsamong the pixels associated with the same single group 131.

The signal control section 444 mainly controls the timing for the A/Dconverter 412 b. The pixel signals output via the output wire 309 passthrough the CDS circuit 412 a and the multiplexer 411 to be input to theA/D converter 412 b. The A/D converter 412 b converts the input pixelsignals into digital signals, under the control of the signal controlsection 444. The pixel signals that have been converted into digitalsignals are handed over to the de-multiplexer 413, and stored as digitaldata pixel values in the pixel memories 414 corresponding respectivelyto the pixels.

The signal processing chip 111 includes a timing memory 430 that servesas an accumulation control memory and stores block classificationinformation indicating which single groups 131 are combined to formblocks and accumulation number information indicating how many timeseach of the blocks has performed the charge accumulation. The timingmemory 430 may be formed by a flash RAM, for example.

In the manner described above, the system control section 501 determineswhich of the single groups are to be combined to form the blocks, basedon the detection results of the scene brightness distribution detectionperformed before the image capturing sequence. The determined blocks areclassified as block 1, block 2, etc., for example, and are defined bywhich of the single groups 131 are included therein. The drive controlsection 420 receives the block classification information from thesystem control section 501, and stores this information in the timingmemory 430.

The system control section 501 determines how many times each blockperforms charge accumulation, based on the detection results of thebrightness distribution. The drive control section 420 receives theaccumulation number information from the system control section 501, andstores this information in the timing memory 430 as a pair with thecorresponding block classification information. By storing the blockclassification information and the accumulation number information inthe timing memory 430 in this manner, the drive control section 420 cansequentially reference the timing memory 430 to independently performthe charge accumulation control. In other words, when an image capturinginstruction signal is received form the system control section 501 whileperforming single image acquisition control, the drive control section420 can finish the accumulation control without later receivinginstructions from the system control section 501 concerning control foreach pixel.

The drive control section 420 receives the updated block classificationinformation and accumulation number information from the system controlsection 501 based on the light measurement result (brightnessdistribution detection result) performed in synchronization with theimage capturing preparation instructions, and suitably updates thecontent stored in the timing memory 430. For example, the drive controlsection 420 updates the timing memory 430 in synchronization with theimage capturing preparation instructions or the image capturinginstructions. With this configuration, higher speed charge accumulationcontrol can be realized, and the system control section 501 can performother process in parallel while the drive control section 420 performsthe charge accumulation control.

The drive control section 420 does not stop at performing the chargeaccumulation control for the imaging chip 113, and also references thetiming memory 430 when performing retrieval control. For example, thedrive control section 420 references the accumulation number informationof each block, and stores the pixel signals output from thede-multiplexer 413 in the corresponding addresses of the pixel memories414.

The drive control section 420 retrieves a target pixel signal from thepixel memory 414 according to a handover request from the system controlsection 501, and hands this pixel signal over to the image processingsection 511. As described above, the pixel memory 414 includes a memoryspace capable of storing pixel signals corresponding to the maximumnumber of integrations for each pixel, and stores the pixel signalscorresponding to the number of accumulations that have been performed aspixel values. For example, in a case where the charge accumulation isperformed 4 times for a given block, since the pixels of this blockoutput pixel signals corresponding to four charge accumulations, fourpixel values are stored in the memory space for each pixel in the pixelmemories 414. When a handover request that requests pixel signals ofidentified pixels is received from the system control section 501, thedrive control section 420 designates the address of the identified pixelin the pixel memory 414, retrieves all of the stored pixel signals, andhands these pixel signals over to the image processing section 511. Forexample, in a case where four pixel values are stored, these four pixelvalues are sequentially handed over, and when only one pixel value isstored, this pixel value is handed over.

The drive control section 420 can retrieve the pixel signals stored inthe pixel memories 414 to the calculating circuit 415, and cause thecalculating circuit 415 to perform the integration process describedabove. The pixel signals that have undergone the integration process arestored in the target pixel addresses of the pixel memories 414. Thetarget pixel addresses may be adjacent in the address space before theintegration process, and may be the same addresses, such that the pixelsignal before the integration process at each address is overwritten.Furthermore, there may be a specialized space for storing together thepixel values of the pixels after the integration process. When ahandover request requesting pixel signals of identified pixels isreceived from the system control section 501, the drive control section420 can hand over the pixel signals after the integration process to theimage processing section 511, according to the specifics of the handoverrequest. Of course, the pixel signals before and after the integrationprocess can be handed over together.

The pixel memories 414 may be provided with a data transfer interfacethat transfers pixel signals according to the handover request. The datatransfer interface is connected to a data transfer wire, which isconnected to the image processing section 511. The data transfer wire isformed by a data bus on a bus line, for example. In this case, thehandover request from the system control section 501 to the drivecontrol section 420 is performed by an address designation using anaddress bus.

The transmission of the pixel signals by the data transfer interface isnot limited to an address designation method, and a variety of methodscan be adopted. For example, when performing a data transfer, a doubledata rate method can be used in which processing is performed using boththe rising and falling edges of a clock signal used to synchronize thecircuits. As another example, a burst transfer method can be used inwhich the data is transferred all at once by omitting a portion of theprocedures, such as address designation, in order to achieve higherspeed. As yet another example, combinations of different methods can beused, such as a combination of a bus method in which the controlsection, the memory section, and the input/output section are connectedin parallel by wiring and a serial method in which data is transferredserially one bit at a time.

With this configuration, the image processing section 511 can receiveonly the necessary pixel signals, and therefore the image processing canbe completed at high speed, especially in a case of forming an imagewith low resolution. Furthermore, when the calculating circuit 415 isperforming the integration process, the image processing section 511need not perform the integration process, and therefore the imageprocessing can be made high speed, by sharing functions and performingparallel processing.

FIG. 11 is a view describing the flow of pixel signals from the imagingchip 113 to the signal processing chip 111. In this example, the singlegroup 131 is formed by 2048 pixels arranged with 32 pixels in each rowand 64 pixels in each column.

In the configuration shown in FIG. 3 , one single group 131 has oneoutput wire 309, but in the example of FIG. 11 , one single group 131has 16 output wires, such that one output wire 309 is shared by pixelsin two adjacent rows. Each output wire 309 branches into two outputwires 309 a and 309 b near the connection portion between the imagingchip 113 and the signal processing chip 111.

On the signal processing chip 111 side, two input wires 409 a and 409 bare provided corresponding respectively to the branched output wires 309a and 309 b. Accordingly, at the connection portion, a bump 109 a isprovided to connect the output wire 309 a and the input wire 409 a, anda bump 109 b is provided to connect the output wire 309 b and the inputwire 409 b. By forming a redundant connection in this manner, thelikelihood of pixel defects occurring due to connection problems can bedecreased. In this sense, the number of branches is not limited to two,and there may be a plurality of branches. Furthermore, the electricalconnection between the output wire and the input wire is not limited toa bump connection, and when adopting an electrical connection in whichconnection problems are possible, the connection section preferably hasa redundant configuration such as described above.

The input wires 409 a and 409 b have switches 461 a and 461 b formedtherein, before input to the CDS circuit 412 a. The switches 461 a and461 b are controlled by the drive control section 420 to move togetheras a switch 461. Specifically, the drive control section 420 usuallyactivates transmission of pixel signals through the input wire 409 a byturning ON the switch 461, and deactivates transmission on the inputwire 409 b by turning OFF the switch 461 b. On the other hand, when itis determined that a connection problem has occurred in the bump 109 a,the drive control section 420 switches OFF the switch 461 a todeactivate the transmission of the input wire 409 a, and switches ON theswitch 461 b to activate the transmission of the input wire 409 b. Inthis way, according to the linked operation of the switch 461 that setsone wire to a connected state and another wire to a disconnected state,it can be expected that only correct pixel signals will be input to theCDS circuit 412 a.

The determination concerning connection problems of the bumps 109 can bemade by the drive control section 420, or can be made by the systemcontrol section 501. The connection problem determination is made basedon the results of the output of pixel signals passed through the inputwire 409. Specifically, the system control section 501 compares pixelsignals output from output wires 309 adjacent on both ends, and when itis determined that there is a difference that is greater than or equalto a predetermined threshold value, determined that there is aconnection problem. A determination standard concerning whether thereare continuous abnormal values in the row direction may also be added.In this case, the drive control section 420 receives the determinationresult of the system control section 501 and switches the switch 461 ofthe corresponding input wire 409. The connection problem judgment may becombined with the image capturing operations and not performedsequentially, and when a judgment is made, the drive control section 420can store the judgment result in the timing memory 430. In this case,the drive control section 420 controls the switch 461 while referencingthe timing memory 430 when retrieving the pixel signals.

In the embodiments described above, the photometric section 503 includesan independent AE sensor. However, the light measurement process can beperformed using a pre-image output from the imaging element 100 beforethe image capturing instructions are received from the user and the mainimage capturing process is performed. Furthermore, the pixels used forlight measurement may be provided in one single group 131, for example.In the pixel arrangement of FIG. 2 , for example, by setting the colorfilter of the pixel in the upper left corner of the single group 131 tobe a transparent filter or not providing a filter, this pixel can beused as a light measurement pixel. Such a pixel can receive visiblelight over a wide bandwidth, and is therefore suitable for detecting thebrightness distribution of a scene. Furthermore, the light measurementpixel may be independent from the single group, and light measurementpixels may be collected to form a new single group. With thisconfiguration, the driving section 502 can independently control thecharge accumulation for the light measurement pixels.

The above embodiments describe an example in which the determinedexposure time T0 matches one charge accumulation time of the shadowregion, but instead, there may be two charge accumulations performed forthe shadow region during the determined exposure time T0, for example.In this case, using the example of FIG. 7 , the shift of the brightnessrange providing a gradation sequence to the high brightness sideaccording to the number of integrations, which is for pixel signals fromthe intermediate region for which the charge accumulation is performedtwice, is not performed for the pixel signals from the shadow region. Inother words, the value of a pixel signal resulting from one chargeaccumulation (charge accumulation time of T0/2) in the shadow region islow, and will not become saturated from a 12-bit range even if twocharge accumulations are added, and therefore there is no need toperform the shift process.

In this way, if a plurality of charge accumulations are set for theshadow region and the adding process is performed for the output pixelsignals, the randomization in the dark portions can be expected to becancelled out. In this case, with the objective of enlarging the dynamicrange, the shift process can be omitted and a simple adding process maybe performed, unlike the processing of the pixel signals in otherregions where the charge accumulation is performed a plurality of times.In the same manner as in the embodiments described above, this type ofprocessing may be performed by the image processing section 511 or bythe calculating circuit 415 provided in the signal processing chip 111.

While the embodiments of the present invention have been described, thetechnical scope of the invention is not limited to the above describedembodiments. It is apparent to persons skilled in the art that variousalterations and improvements can be added to the above-describedembodiments. It is also apparent from the scope of the claims that theembodiments added with such alterations or improvements can be includedin the technical scope of the invention.

The operations, procedures, steps, and stages of each process performedby an apparatus, system, program, and method shown in the claims,embodiments, or diagrams can be performed in any order as long as theorder is not indicated by “prior to,” “before,” or the like and as longas the output from a previous process is not used in a later process.Even if the process flow is described using phrases such as “first” or“next” in the claims, embodiments, or diagrams, it does not necessarilymean that the process must be performed in this order.

LIST OF REFERENCE NUMERALS

-   -   100: imaging element, 101: microlens, 102: color filter, 103:        passivation film, 104: PD (Photo Diode), 105: transistor, 106:        PD layer, 107: wire, 108: wiring layer, 109: bump, 110: TSV        (Through Silicon Via), 111: signal processing chip, 112: memory        chip, 113: imaging chip, 131: single group, 302: transfer        transistor, 303: reset transistor, 304: amplification        transistor, 305: selection transistor, 306: reset wire, 307: TX        wire, 308: decoder wire, 309: output wire, 310: Vdd wire, 311:        negative current source, 409: input wire, 411: multiplexer, 412:        signal processing circuit, 413: de-multiplexer, 414: pixel        memory, 415: calculating circuit, 420: drive control section,        430: timing memory, 441: sensor control section, 442: block        control section, 443: synchronization control section, 444:        signal control section, 461: switch, 500: imaging apparatus,        501: system control section, 502: driving section, 503:        photometric section, 504: work memory, 505: storage section,        506: display section, 511: image processing section, 512:        calculating section, 601: shadowed subject, 602: intermediate        subject, 603: highlighted subject, 604: frame, 611: shadow        region, 612: intermediate region, 613: highlight region

What is claimed is:
 1. An imaging sensor comprising: a firstsemiconductor substrate having a plurality of pixels; and a secondsemiconductor substrate having a driving section that outputs a firstcontrol signal for controlling at least a first pixel among theplurality of pixels, and a second control signal for controlling atleast a second pixel among the plurality of pixels.